-
Notifications
You must be signed in to change notification settings - Fork 456
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add metrics datastream for Azure Functions #7130
Conversation
🌐 Coverage report
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
First pass is good, there are a few details to discuss.
- name: eventhub | ||
type: text | ||
title: Event Hub | ||
multi: false | ||
required: true | ||
show_user: true | ||
description: >- | ||
Elastic recommends using one event hub for each integration. Visit [Create an event hub](https://docs.elastic.co/integrations/azure#create-an-event-hub) to learn more. Use event hub names up to 30 characters long to avoid compatibility issues. | ||
- name: consumer_group | ||
type: text | ||
title: Consumer Group | ||
multi: false | ||
required: true | ||
show_user: true | ||
default: $Default | ||
- name: connection_string | ||
type: password | ||
title: Connection String | ||
multi: false | ||
required: true | ||
show_user: true | ||
description: >- | ||
The connection string required to communicate with Event Hubs. See [Get an Event Hubs connection string](https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-get-connection-string) to learn more. | ||
- name: storage_account | ||
type: text | ||
title: Storage Account | ||
multi: false | ||
required: true | ||
show_user: true | ||
description: >- | ||
The name of the storage account where the consumer group's state/offsets will be stored and updated. | ||
- name: storage_account_key | ||
type: password | ||
title: Storage Account Key | ||
multi: false | ||
required: true | ||
show_user: true | ||
description: >- | ||
The storage account key, this key will be used to authorize access to data in your storage account. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since we are moving these vars from the "integration package scope" to the "data stream scope", we should test the upgrade path from 0.1.0 to the new version.
I know this is integration is experimental, but we should see what's the user experience in this use case.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I tried to test the upgrade from 0.0.1 to a new version, including the installations of assets. I was able to ingest logs after the upgrade in ES. So, this should be good enough to go ahead.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is cool.
Did the upgrade kept all the integration settings values like event hub name, connection string, etc. from the integration scope to data stream scope?
packages/azure_functions/data_stream/metrics/elasticsearch/ingest_pipeline/default.yml
Outdated
Show resolved
Hide resolved
packages/azure_functions/data_stream/metrics/elasticsearch/ingest_pipeline/default.yml
Outdated
Show resolved
Hide resolved
- name: domain | ||
level: extended | ||
type: keyword | ||
ignore_above: 1024 | ||
description: 'Name of the domain of which the host is a member. | ||
|
||
For example, on Windows this could be the host''s Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host''s LDAP provider.' | ||
example: CONTOSO | ||
default_field: false | ||
- name: hostname | ||
level: core | ||
type: keyword | ||
ignore_above: 1024 | ||
description: 'Hostname of the host. | ||
|
||
It normally contains what the `hostname` command returns on the host machine.' | ||
- name: id | ||
level: core | ||
type: keyword | ||
ignore_above: 1024 | ||
description: 'Unique host id. | ||
|
||
As hostname is not always unique, use values that are meaningful in your environment. | ||
|
||
Example: The current usage of `beat.name`.' | ||
- name: ip | ||
level: core | ||
type: ip | ||
description: Host ip addresses. | ||
- name: mac | ||
level: core | ||
type: keyword | ||
ignore_above: 1024 | ||
description: Host mac addresses. | ||
- name: name | ||
level: core | ||
type: keyword | ||
ignore_above: 1024 | ||
description: 'Name of the host. | ||
|
||
It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use.' | ||
- name: os.family | ||
level: extended | ||
type: keyword | ||
ignore_above: 1024 | ||
description: OS family (such as redhat, debian, freebsd, windows). | ||
example: debian | ||
- name: os.kernel | ||
level: extended | ||
type: keyword | ||
ignore_above: 1024 | ||
description: Operating system kernel version as a raw string. | ||
example: 4.4.0-112-generic | ||
- name: os.name | ||
level: extended | ||
type: keyword | ||
ignore_above: 1024 | ||
multi_fields: | ||
- name: text | ||
type: text | ||
norms: false | ||
default_field: false | ||
description: Operating system name, without the version. | ||
example: Mac OS X | ||
- name: os.platform | ||
level: extended | ||
type: keyword | ||
ignore_above: 1024 | ||
description: Operating system platform (such centos, ubuntu, windows). | ||
example: darwin | ||
- name: os.version | ||
level: extended | ||
type: keyword | ||
ignore_above: 1024 | ||
description: Operating system version as a raw string. | ||
example: 10.14.1 | ||
- name: type | ||
level: core | ||
type: keyword | ||
ignore_above: 1024 | ||
description: 'Type of host. | ||
|
||
For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment.' | ||
- name: containerized | ||
type: boolean | ||
description: > | ||
If the host is a container. | ||
|
||
- name: os.build | ||
type: keyword | ||
example: "18D109" | ||
description: > | ||
OS build information. | ||
|
||
- name: os.codename | ||
type: keyword | ||
example: "stretch" | ||
description: > | ||
OS codename, if any. | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should check if we can leverage ECS for these fields definition.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we switch to the ECS definition for these fields? They look the same.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Almost there!
Just a couple of double-checks and a nit:
- double-check: Did the upgrade kept all the integration settings values like event hub name, connection string, etc. from the integration scope to data stream scope?
- double-check: Can we use ECS for fields definitions at
packages/azure_functions/data_stream/metrics/fields/fields.yml
? - nit: what about moving credentials at the top of the data stream settings?
- name: eventhub | ||
type: text | ||
title: Event Hub | ||
multi: false | ||
required: true | ||
show_user: true | ||
description: >- | ||
Elastic recommends using one event hub for each integration. Visit [Create an event hub](https://docs.elastic.co/integrations/azure#create-an-event-hub) to learn more. Use event hub names up to 30 characters long to avoid compatibility issues. | ||
- name: consumer_group | ||
type: text | ||
title: Consumer Group | ||
multi: false | ||
required: true | ||
show_user: true | ||
default: $Default | ||
- name: connection_string | ||
type: password | ||
title: Connection String | ||
multi: false | ||
required: true | ||
show_user: true | ||
description: >- | ||
The connection string required to communicate with Event Hubs. See [Get an Event Hubs connection string](https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-get-connection-string) to learn more. | ||
- name: storage_account | ||
type: text | ||
title: Storage Account | ||
multi: false | ||
required: true | ||
show_user: true | ||
description: >- | ||
The name of the storage account where the consumer group's state/offsets will be stored and updated. | ||
- name: storage_account_key | ||
type: password | ||
title: Storage Account Key | ||
multi: false | ||
required: true | ||
show_user: true | ||
description: >- | ||
The storage account key, this key will be used to authorize access to data in your storage account. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is cool.
Did the upgrade kept all the integration settings values like event hub name, connection string, etc. from the integration scope to data stream scope?
packages/azure_functions/data_stream/metrics/elasticsearch/ingest_pipeline/default.yml
Outdated
Show resolved
Hide resolved
- name: domain | ||
level: extended | ||
type: keyword | ||
ignore_above: 1024 | ||
description: 'Name of the domain of which the host is a member. | ||
|
||
For example, on Windows this could be the host''s Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host''s LDAP provider.' | ||
example: CONTOSO | ||
default_field: false | ||
- name: hostname | ||
level: core | ||
type: keyword | ||
ignore_above: 1024 | ||
description: 'Hostname of the host. | ||
|
||
It normally contains what the `hostname` command returns on the host machine.' | ||
- name: id | ||
level: core | ||
type: keyword | ||
ignore_above: 1024 | ||
description: 'Unique host id. | ||
|
||
As hostname is not always unique, use values that are meaningful in your environment. | ||
|
||
Example: The current usage of `beat.name`.' | ||
- name: ip | ||
level: core | ||
type: ip | ||
description: Host ip addresses. | ||
- name: mac | ||
level: core | ||
type: keyword | ||
ignore_above: 1024 | ||
description: Host mac addresses. | ||
- name: name | ||
level: core | ||
type: keyword | ||
ignore_above: 1024 | ||
description: 'Name of the host. | ||
|
||
It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use.' | ||
- name: os.family | ||
level: extended | ||
type: keyword | ||
ignore_above: 1024 | ||
description: OS family (such as redhat, debian, freebsd, windows). | ||
example: debian | ||
- name: os.kernel | ||
level: extended | ||
type: keyword | ||
ignore_above: 1024 | ||
description: Operating system kernel version as a raw string. | ||
example: 4.4.0-112-generic | ||
- name: os.name | ||
level: extended | ||
type: keyword | ||
ignore_above: 1024 | ||
multi_fields: | ||
- name: text | ||
type: text | ||
norms: false | ||
default_field: false | ||
description: Operating system name, without the version. | ||
example: Mac OS X | ||
- name: os.platform | ||
level: extended | ||
type: keyword | ||
ignore_above: 1024 | ||
description: Operating system platform (such centos, ubuntu, windows). | ||
example: darwin | ||
- name: os.version | ||
level: extended | ||
type: keyword | ||
ignore_above: 1024 | ||
description: Operating system version as a raw string. | ||
example: 10.14.1 | ||
- name: type | ||
level: core | ||
type: keyword | ||
ignore_above: 1024 | ||
description: 'Type of host. | ||
|
||
For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment.' | ||
- name: containerized | ||
type: boolean | ||
description: > | ||
If the host is a container. | ||
|
||
- name: os.build | ||
type: keyword | ||
example: "18D109" | ||
description: > | ||
OS build information. | ||
|
||
- name: os.codename | ||
type: keyword | ||
example: "stretch" | ||
description: > | ||
OS codename, if any. | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we switch to the ECS definition for these fields? They look the same.
Can we upload a sample event to cross-check? |
I ran a quick test setting up 0.0.1 with logs ingestion and then upgrading to 0.1.1. The upgrade triggers the fields conflict flow, and by default users must enter event hub and storage account details again. CleanShot.2023-08-16.at.12.46.45.mp4 |
Conflict management like this is fine per se. It's the expected behavior when we remove or move the vars from one scope (global) to another (data stream). @lalit-satapathy, can we somehow hint at the UI of picking the values for the old vars? In general, how do we usually handle situations like this? We just let it be, or mention this in the docs? I never faced an integration conflict case before. |
Not sure, why this has happened. Was is trigged by the moving of the variables to the data stream scope #7130 (comment) ? @zmoog, May be not much use of this 0.0.1 package currently? So, if the change in itself is good (Please confirm that), it will be less impact. |
Yep, with commit 843df5f we moved the vars from: packages/azure_functions/manifest.yml (package scope) Into: packages/azure_functions/data_stream/functionapplogs/manifest.yml (integration / data stream scope) The main reason for this change is the This problem will eventually go away when we upgrade the
The changes in this PR are good. The variables conflict management is working as expected, and since this is a technical preview, I believe it's OK to go through this flow. If there are reasons to ship this version as we are ready, the other points in the previous comment are nits we can address later. |
For the records, @devamanv also put together a few other observations about differences in hosting plan options. |
|
||
### Metrics | ||
**Metrics** give you insight into the state of your Azure costs. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The description here is not relevant to the azure function metrics. Can you please remove the cost specific information and add appropriate description.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Removed the unwanted details from the description.
processors: | ||
- set: | ||
field: ecs.version | ||
value: "8.0.0" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Update the ECS version to the latest stable version. Also, make sure both the logs and metrics datastream have a sets the same ECS version.
| azure.functions.bytes_sent.total | The amount of outgoing bandwidth consumed by the app, in MiB. | long | | ||
| azure.functions.current_assemblies.average | The current number of Assemblies loaded across all AppDomains in this application. | long | | ||
| azure.functions.file_system_usage.average | Percentage of filesystem quota consumed by the app. | long | | ||
| azure.functions.function_execution_count.total | Function Execution Count. For FunctionApps only. | long | |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add metric_type for the applicable fields.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added metric_type as well as unit wherever applicable.
@@ -6,19 +6,22 @@ Use this integration to build web APIs, respond to database changes, process IoT | |||
|
|||
|
|||
## Data streams | |||
The Azure Functions integration contains two data streams: [Logs](#functionapplogs) and [Metrics](#metrics) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are you linking any references in the logs and metrics ? I don't see a link attached to it. I see this is done to reference the title below but literally, when clicking the link I am not able to go the metrics section.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I fixed the link, now it correctly navigates to the metrics section.
Hey @muthu-mps, thank you for taking the time to read this in detail! I missed the leftovers from the Azure Billing setup information we used as the template. @devamanv, we need to update the docs specifying the role with the least privileges to access Azure Monitor. |
I have pushed the changes to only keep the Functions specific information in the docs. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A few nits.
However, since this is a technology preview integration, if we want to ship this to gather feedback, we can address them later.
@@ -1,4 +1,9 @@ | |||
# newer versions go on top | |||
- version: "0.1.0" | |||
changes: | |||
- description: Add Azure Functions metrics data stream and a Kibana dashboard |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The datastream comes with a dashboard. We can remove the and a Kibana dashboard
from changelog entry.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I added it to address one of the review comments. However, since a datastream is always expected to have a dashboard, we can remove the explicit mention of it.
external: ecs | ||
- name: service.address | ||
type: keyword | ||
description: Service address |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think there is no need to add description
for ECS fields as they are generated at build time. Also this would override the already present description of the ECS field.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@aliabbas-elastic - The next question is why we need to define a type
for the ECS field. @devamanv we need to reference the ECS field here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for pointing this out. Yes, we should remove the description
and the type
fields, and add the missing external
field. Will make this change.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@muthu-mps, Is there a ECS field mapping guideline document, which captures the correct way to map ECS fields?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@lalit-satapathy - I don't find a guidelines document for mapping ECS fields. I do see the service.address
is mapped directly in some and referencing ECS in some integrations.
/test |
...s/azure_functions/kibana/dashboard/azure_functions-5b40c9c0-33d4-11ee-8d85-2d7adebebd1b.json
Outdated
Show resolved
Hide resolved
"type": "Microsoft.Web/sites", | ||
"group": "test-rg", | ||
"tags": { | ||
"hidden-link: /app-insights-resource-id": "/subscriptions/12hjkls-78tyu-404f-a3d2-1dc9982f45ds/resourceGroups/test-rg/providers/Microsoft.Insights/components/return-of-the-jedi" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you create custom tags and associate them with this function and get the metrics out of it?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These tags are coming from Azure monitor itself. To add custom tags, we will have to add a new field for tags and conditionals to the agent configuration to set it. I can give that a try but could be taken as an enhancement. I will create a GH issue for it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As the tags
field is dynamic, not necessary to add a new field. You can just try adding custom tags to the functions and check if the key values are captured. Ex: tags.*
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks Good!
Package azure_functions - 0.1.0 containing this change is available at https://epr.elastic.co/search?package=azure_functions |
What does this PR do?
The PR contains changes to add metrics datastream for Azure Functions.
Checklist
changelog.yml
file.How to test this PR locally
elastic-package
elastic-package test
from the integrations/packages/azure_functions directoryRelated issues
Implements a part of #6704
Screenshots